| Document status | 35 - Reviewed |
|---|
File structure of TimeSeries data
One container per DSO, which makes sures that DSO 1, can't read, write or delete DSO 2's files.
DSO 1 Blob Container
|
├── TimeSeries/
| |
│ ├── TimeResolution=Hourly/
| | |
│ │ ├── Quadrant=Consumption/
│ │ │ ├── 2025-10-30.parquet
│ │ │ └── 2025-10-31.parquet
| | |
│ │ └── Quadrant=Production/
│ │ ├── 2025-10-30.parquet
│ │ └── 2025-10-31.parquet
| |
│ └── TimeResolution=Quarterly/
| |
│ ├── Quadrant=Consumption/
│ │ ├── 2025-10-30.parquet
│ │ └── 2025-10-31.parquet
| |
│ └── Quadrant=Production/
│ ├── 2025-10-30.parquet
│ └── 2025-10-31.parquet
|
└── MetaData/
├── MeterMetaData.parquet
├── ConsumptionPeaks.parquet
└── ProductionPeaks.parquet
TimeSeries Folder
Contains the timeseries data used by the Machine Learning, sorted into subfolders to identify what type.
- TimeResolution=Hourly: Hourly timeseries - Initialy needs a years worth of data.
- TimeResolution=Quarterly: Quarter hour timeseries. Initially needs 6 weeks worth of data.
Both contains the following sub folders:
- Quadrant=Consumption: Timeseries related to the Consumption / A14 timeseries.
- Quadrant=Production: Timeseries related to the Production / A23 timeseries.
The final files only contain one days worth of data, and is named after that day, like: 2025-10-31.parquet
This improves the speed of which to the files can be read.
What is Hourly timeseries and what is Quarter hourly timeseries?
A hourly timeseries is understood as a timeseries where an quantity at a timestamp is the quantity the following hour. A data point with reading_ts 12:00 describes the usage from 12:00 to 13:00. A quarter hour timeseries is understood as a timeseries where an quantity at a timestamp is the quantity the following quarter hour. A data point with reading_ts 14:00 describes the usages from 14:00 to 14:15.
File format
All data is stored in Parquet files. Read more here
To minimize file size and read speed, order the data by meter_id then reading_ts.
| Property Name | Datatype | Comment |
|---|---|---|
| meter_id | string (VARCHAR) | Unique Id that identifies that Meter. |
| reading_ts | DateTime YYYY-MM-DDThh:mm:ss (TIMESTAMP) | In UTC. The Timestamp for the beginning of the reading period of the delta value. I.E. On the 2nd of November 2025, from 15:00 to 15:15 was used 0,15 kWh, the reading_ts would be 2025-11-02T15:00:00 |
| converted_quantity | Double (DOUBLE) | In kWh. Calculated with a potential converstion factor. Is either in a hourly resolution or a quarter hour resolution depended on subfolder placement. If value is missing, preferably skip row, or set value to null. |
| quadrant | Integer (INTEGER) | 1 = Consumption / A14 2 = Production / A23 |
Meta Data Folder
This folder contains meta data about the meters used to help filtering of friends and improve estimation. It contains 3 files:
- MeterMetaData.parquet
- ConsumptionPeaks.parquet
- ProductionPeaks.parquet
Meter Meta Data
The meter meta data file contains addition data used by the estimation system.
Each row indicates a meter and a period in which the meta data applies.
A meter can have multiple active periods across multiply rows. This happens if the meter is moved or other changes is made to it's meta data.
One parquet file, sorted by meter_id, containing the following:
| Property Name | Datatype | Comment |
|---|---|---|
| meter_id | string (VARCHAR) | Unique Id that identifies that Meter. |
| commissioning_date | DateTime YYYY-MM-DDThh:mm:ss (TIMESTAMP) | In UTC. The date on which the meter was put up and activated. The active period is the dates in the which the following meta data is correct. |
| decommissioning_date | DateTime YYYY-MM-DDThh:mm:ss (TIMESTAMP) | In UTC. The date on which the meter was taken down and deactivated. |
| grid_connection_level | string (VARCHAR) | Grid Connection Level / Placement in the Network. - AHigh - ALow - BHigh - BLow - C - AHighPlus - A0 - NotSpecified |
Timeseries Peaks
Two files containing the peaks of the meters.
One for Consumption peaks and one for Production peaks.
Data scan is limited to 2 years.
One parquet file, sorted by meter_id, containing the following:
| Property Name | Datatype | Comment |
|---|---|---|
| meter_id | Integer (INTEGER) | Unique Id that identifies that Meter |
| quadrant | Integer (INTEGER) | 1 = Consumption / A14 2 = Production / A23 |
| peak_converted_quantity | Double (DOUBLE) | In kWh. The highest delta value recorded on that meter in the last 2 years. |